Language model pretraining generally targets a broad range of use cases and incorporates data from diverse sources. However, there are instances where we desire a model that excels in specific areas without markedly compromising performance in other areas. A cost-effective and straightforward approach is sampling with low-dimensional data features, which allows to select large-scale pretraining data for domain-specific use cases. In this work, we revisit importance sampling with n-gram features consisting of multi-granular tokens, which strikes a good balance between sentence compression and representation capabilities. We observed the sampled data to have a high correlation with the target downstream task performance while preserving its effectiveness on other tasks. This leads to the proposed data sampling paradigm where language models can be pretrained more efficiently on selected documents. On eight benchmarks we demonstrate with $\sim$1% of the data, pretrained models perform on par with the full RefinedWeb data and outperform randomly selected samples for model sizes ranging from 125M to 1.5B.

In a number of situations, collecting a function value for every data point may be prohibitively expensive, and random sampling ignores any structure in the underlying data. We introduce a scalable optimization algorithm with no correction steps (in contrast to Frank-Wolfe and its variants), a variant of gradient ascent for coreset selection in graphs, that greedily selects a weighted subset of vertices that are deemed most important to sample. Our algorithm estimates the mean of the function by taking a weighted sum only at these vertices, and we provably bound the estimation error in terms of the location and weights of the selected vertices in the graph. In addition, we consider the case where nodes have different selection costs and provide bounds on the quality of the low-cost selected coresets. We demonstrate the benefits of our algorithm on point clouds and structured graphs, as well as sensor placement where the cost of placing sensors depends on the location of the placement. We also elucidate that the empirical convergence of our proposed method is faster than random selection and various clustering methods while still respecting sensor placement cost. The paper concludes with validation of the developed algorithm on both synthetic and real datasets, demonstrating that it performs very well compared to the current state of the art.

High dimensionality and class imbalance are two important concerns when training tweet sentiment classifiers. Feature selection techniques reduce dimensionality by selecting an optimal subset of features. Class imbalance can be addressed by either using classifiers that are robust to the impact of class imbalance, such as those trained with an ensemble learning technique, or by using data sampling techniques to create a sampled training set with a more balanced class ratio. These separate techniques can be combined together to address both class imbalance and high-dimensionality; however, it is unclear if it is necessary to use data sampling and ensemble techniques together as both are used to target class imbalance. In our study, we investigate if the addition of random undersampling to Select-Boost (feature selection and boosting) significantly improves the performance of sentiment classifiers trained on imbalanced tweet data. We evaluate classifiers trained using four base learners and three feature subset sizes across two highly dimensional imbalanced datasets. Our results show, for tweet sentiment, the inclusion of random undersampling significantly improves classification performance and indicates this may be more noticeable on datasets with greater levels of class imbalance.